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(54) Precision spatial mapping with combined video and infrared signals 



(57) A system (10) for virtually modeling a physical 
system having immovable and movable objects in- 
cludes at least two video cameras (20-24, 62), each of 
the video cameras being configured to provide a se- 
quence of images. An image processing system (60) ex- 
tracts modulated infrared signals from the sequence of 
images to identify the spatial location of objects (55, 70, 



75) using information obtained from tooth visible light im- 
ages and inf rared images. With this irnformation, a virtual 
reality modeling system constructs aa virtual reality mod- 
el. Infrared pointers that direct modulated infrared spots 
having a unique identification agaimst surfaces can be 
used for surveying, while active (50,, 57) or passive (51) 
infrared tags can track movable objeects for virtual reality 
modeling. 
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Description 

The present invention relates to a system using in- 
frared signaling devices to map areas and track objects. 
More specifically, the present invention relates to the s 
use of CCD video cameras for detecting and identifying 
multiple colocated tagged regions or objects with reflec- 
tive, active, or passive infrared signals, and determining 
spatial location of the identified region or object after im- 
age analysis of video frames generated by the CCD vid- w 
eo camera. 

The graphics and processing capabilities of availa- 
ble personal computers and workstations now permits 
display of relatively sophisticated virtual reality con- 
structions. Currently, the majority of virtual reality con- 
structions are probably imaginary constructs, imple- 
mented as games, simulations, or recreational online 
meeting places, and designed using virtual reality mod- 
eling languages or the equivalent. Although these con- 
structs may be based on a physical model (e.g. rooms 20 
in a buildings), they do not typically employ sensor sys- 
tems to monitor changes in the physical model, and ac- 
cordingly change the virtual reality construct to match 
the physical model. 

One possible reason for the limited availability of vir- 25 
tual reality constructs that accurately track physical 
models is the cost associated with mapping and survey- 
ing the physical model. If the physical system is well de- 
fined and compact, a limited number of sensors or sev- 
eral tracking cameras may suffice to track the state of 30 
the physical system. Such limited physical systems may 
include, for example, the notorious internet accessible 
coffee makers or soda dispensing machines having a 
physical state that can be monitored and displayed. Un- 
fortunately, the sensor or camera systems used to track 
such simple physical systems do not scale very well to 
more useful virtual reality constructs. For example, mod- 
eling a building with sufficient accuracy to track opening 
and closing of doors or windows, movement of objects 
or people, and placement of movable furniture in the 
building requires essentially continuous input of large 
amounts of 3-dimensional spatial information that is not 
readily available with current sensing techniques. 

The problem of dealing with the vast amounts of 
sensor data needed to provide an accurate virtual reality 
model of a building or other complex collection of phys- 
ical objects can be reduced by conceptually dividing the 
collection of physical objects into immovable objects 
and movable objects. In a building, immovable objects 
would include the building's floors, walls, ceilings, over- 
head lights or other permanent fixtures that are infre- 
quently or never moved. Immovable objects need only 
be mapped once with high precision, and require only 
limited long term sensor tracking, which can be option- 
ally updated on a daily, weekly, or monthly time scale to 
ensure that movement of cubicles, partitions, etc. has 
not occurred. Although the initial investment in mapping 
immovable objects for translation to a virtual reality mod- 



el is substantial, maintenance of an accurate virtual re- 
ality model for immovable objects is* not overly difficult. 

In contrast, one of the most difficcult problems asso- 
ciated with virtual reality modeling i is that of tracking 
readily movable objects. Trackable cobjects may include 
books, tools, doors, windows, porUable computers, or 
even people. Doors and windows can be opened or 
closed, books can be misplaced or' incorrectly filed on 
shelves, portable computers can toe moved to other 
rooms, and tools can be lost in a junmble of material in a 
corner of a room. One possible solution to this problem 
relies on tagging an object with a lowv power transmitter 
capable of emitting an identification signal. With such a 
system, the presence or absence off a particular tagged 
object can be ascertained. Unfortunately, the spatial 
resolution of generally available identification systems 
is quite low, with localization only on the order of several 
meters (i.e. room sized) being posssible. Further, such 
systems can generally only track a lirmited number of ob- 
jects, may only be able to provide serial identification 
and localization (instead of substantially parallel identi- 
fication and localization), and often i require proximity of 
the detector (for example, bar code reading systems or 
implanted identification devices for ttagging pets or lab- 
oratory animals). While sophisticated or custom de- 
signed systems that support both cobject identification 
and precision localization are available, these solutions 
are generally quite expensive or hav/e a limited range of 
applications. 

What is needed is an inexpensiwe system mapping 
spatial regions with high precisioni. Once mapped, a 
spatial region should support identiffication and tracking 
of tagged objects without requiring pjroximity of a detec- 
tor. The system should support ideentification of large 
numbers of mapping points, tagged • objects, or regions, 
and have a spatial localization resolution on the order 
of centimeters or millimeters, rather tthan meters, to per- 
mit identification and accurate spatial localization within 
less than room size regions. Such ia system would re- 
quire only inexpensive componemts such as CCD 
(charge couple device) video cameras to act a detec- 
tors, inexpensive passive or active imfrared devices that 
function as surveying points or idemtification tags, and 
an image processing computer for determining identifi- 
cation and spatial localization of a ttagged object or re- 
gion. Ideally, such a system would also have the ability 
to track and spatially localize slow rmoving objects (e.g. 
a tagged person or electronic device held by a person). 

The present invention provides; a system for virtu- 
ally modeling a physical system hav/ing immovable and 
movable objects, comprising: at leasst two video camer- 
as, each of the video cameras beinog configured to pro- 
vide a sequence of images, an ima»ge processing sys- 
tem configured to extract modulatted infrared signals 
from the sequence of images and identify the spatial lo- 
cation of objects, and a virtual realitty modeling system 
for constructing a virtual reality moodel using identified 
object spatial location information. 
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The invention further provides a system according 
to claims 4 and 7 of the appended claims. 

In one preferred embodiment the present invention 
provides a system for precisely locating infrared tags 
useful for mapping and surveying purposes. At least two $ 
conventional and widely commercially available CCD 
video cameras having overlapping field of views are 
used. The video cameras are capable of detecting both 
visible light and infrared, and are further configured to 
provide a sequence of images at predefined frame 10 
rates. One or more infrared identification tags for pro- 
viding modulated infrared signals are positioned in a 
room or area within the overlapping field of view of the 
video cameras, and an image processing system is 
used to extract the modulated infrared signals from the is 
sequence of images and identify the spatial location of 
the infrared tag using information obtained from both 
visible light images and infrared images. In certain pre- 
ferred embodiments, reflective tagging systems using 
laser pointers to direct uniquely identifiable light spots 20 
against predefined regions or areas of a room are best 
suited for surveying. 

Various possible infrared signaling modes of oper- 
ation are contemplated. In one preferred embodiment 
infrared identification tags attached to a particu lar region 25 
or object intermittently emit infrared detection signals at 
a rate less than the frame rate of the video camera to 
establish spatial location. The pattern of infrared blink- 
ing seen through comparison of multiple frames of the 
video cameras can be used to positively identify infrared 
tags and transfer identification information or other data. 
Advantageously, because multiple infrared identifica- 
tion tags are spatially separated within the same frame 
of the video camera, identification and tracking of mul- 
tiple tags in parallel can be achieved. In practice, data 
acquisition from a room can utilize large numbers of 
identifying infrared tags attached to objects. In addition 
to objects, infrared tags can be deployed to describe 
particular positions, locations, or areas in a room, or can 
be used as remote sensors for providing telemetry and 
location dependent data (e.g. thermometers or motion 
detectors). 

In certain embodiments, multiple tags attached to a 
movable object can be used to determine and track ori- 
entation, as well as location, of the objects. For example, 
if an object is rotated, use of two or more attached tags 
could allow determination of the angle of rotation. Use 
of multiple tags also alleviates problems associated with 
obscuration of an object, and increases accuracy of po- 
sition determination. Further, multiple identification tags 
can be used to increase rate of data transfer. 

Alternative embodiments, including the use of visi- 
ble light instead of infrared, or variant position and data 
encoding schemes for infrared signals, are also contem- 
plated within the scope of the present invention. For ex- 
ample, it is possible to use a separate infrared commu- 
nication channel receiver for reception of infrared iden- 
tification signals that are emitted substantially time co- 



incident with infrared detection signaals used to establish 
spatial location of the infrared tag. Using time coinci- 
dence methods, it is possible to trainsfer data at much 
higher rates than the relatively slow/ data transfer rates 
possible using multiple frame comparisons from the vid- 
eo cameras alone. In addition, to rectluce power require- 
ments it is possible to use passive irnfrared reflectors as 
tags (with a separate, room mountted infrared flasher 
providing infrared light on demand)), or to use infrared 
identification tags that are activated only in response to 
an identification request. 

In a broad embodiment, the preesent invention is a 
system for precisely locating identification tags that in- 
cludes at least two video cameras configured to provide 
a sequence of images at a first frarme rate. An identifi- 
cation tag for providing a series of sppaced apart signals 
at a second frame rate defined to b»e less than the first 
frame rate, with the signals being perceptible by the vid- 
eo camera. An image processing swstem is configured 
to extract the three dimensional spsatial location of the 
identification tags from the sequence of images, and a 
data transfer system is used for tramsferring data from 
the identification tag. This data is (correlated with the 
three dimensional spatial location cof the identification 
tag as determined by the image proocessing system. 

Advantageously, the present imvention uses rela- 
tively inexpensive and commonly av?ailable components 
such as infrared transmitters and CJCD video cameras 
to provide highly precise spatial localization of identifi- 
cation tags. Suitable high quality C/CD video cameras 
are widely employed for consumerr security systems. 
Because of the economies of scalee, such CCD video 
cameras are quite inexpensive, maiking them ideal for 
this application. 

Since infrared signals are invisible to the human 
eye, yet easily visible to CCD video cameras, the infra- 
red signaling system is essentially invisible to users, 
while still being easy for automated systems to locate 
and interpret without elaborate image processing tech- 
niques. The present invention cam be quickly imple- 
mented after computer network comnected video cam- 
eras are installed, and does not require complex setup 
or initialization. Using a limited nurmber of inexpensive 
video cameras, the present inventiorn allows for tracking 
large numbers of infrared tags capable of conveying lim- 
ited amounts of object data, with density bandwidth 
problems typical of cellular mobile ccommunications be- 
ing substantially avoided. 

Additional functions, objects, advantages, and fea- 
tures of the present invention will become apparent from 
consideration of the following description and drawings 
of preferred embodiments, in which:: 

Figure 1 is a schematic outline* of system for pre- 
cisely locating infrared identification tags using a 
CCD video camera; 

Figure 2 is an expanded view of a search region 
illustrated in Figure 1 , with boobks having single or 
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multiple infrared tags being illustrated; 
Figure 3 is a graph illustrating infrared pulse inten- 
sity versus time, with periodic pulses to establish 
position followed by data pulses to transfer informa- 
tion being shown; 5 
Figure 4 illustrates selected image processed 
frames of two video cameras showing blinking in- 
frared data signals capable of being correlated to 
determine spatial position; 

Figure 5 is an alternative pulse scheme using two w 
substantially time coincident communication chan- 
nels to determine position and transfer data; 
Figure 6 illustrates selected image processed 
frames of two video cameras operating in accord- 
ance with the alternative pulse scheme of Figure 5, is 
with blinking infrared data signals capable of being 
correlated to determine spatial position on a first 
communication channel, and data provided on a 
substantially time coincident second communica- 
tion channel; and 20 
Figure 7 schematically illustrates construction of an 
active infrared tag. 

As illustrated in Figure 1, a system 10 for precisely 
locating infrared signal sources 25 (infrared tags) in- 25 
eludes multiple CCD video cameras 25 positioned in 
room 12. These video cameras 25 may have a fixed 
view, such as CCD video cameras 20, 21, 22, and 62, 
or they may have a movable view such as provided by 
movable CCD video camera 24. Infrared signal sources 30 
45 suitable for spatial localization, identification and da- 
ta transference can be positioned on static or essentially 
immovable objects such as walls (tags 40, 41 , and 42), 
or desks (tag 44). Infrared signal sources 45 can also 
be positioned on readily movable objects such as books 35 
55 (tags 50 and 51 ), a portable electronic device 70 (tag 
72), or even writing instruments such as pen 75 (tags 
76 and 77). Image processing for spatial localization and 
data utilizes a computer system, in this example illus- 
trated by a computer system 60. The computer system 
60 can of course be located outside room 1 2, and there 
is no special requirement for any local computer process 
control. The computer system is connected by wireless 
or wired links to video cameras 25, and may be stand- 
alone or connected to a computer network for distribut- 45 
ing image processing duties and allowing for high speed 
data transfer. In the illustrated embodiment of Figure 1, 
the computer system 60 has sufficient image processing 
capabilities and data storage to construct a virtual reality 
model of the room 12, with all objects tagged with an so 
infrared signal source being trackable. 

The present system 10 advantageously utilizes 
emitted infrared signals that are invisible to the human 
eye yet easily visible to CCD cameras 25. After suitable 
image processing, the emitted infrared signals from mul- ss 
tiple infrared tags provide three dimensional spatial lo- 
calization for each of those multiple infrared tags. The 
emitted infrared signals are typically intermittent point 



source flashes of infrared light (infrared blinks) that ap- 
pear, along with the visual light image, on frames of CCD 
video cameras. Since the cameras I25 typically provide 
between about 1 0 to 30 frames per ssecond of image da- 
ta, the blink rate is selected to be less than about 5 to 
15 blinks per second to ensure capture of infrared im- 
ages without aliasing problems. Sub-tractive image 
processing between adjacent image frames or other 
suitable image processing techniques is used to enhance 
and separate the infrared signal frcom the visual back- 
ground, allowing the two dimensional spatial pattern of 
infrared blinks in each image processsed frame to be de- 
termined. Advantageously, the image processing tech- 
niques required to enhance and differentiate infrared 
point sources in a predominantly vissible light image for 
operation of the present system cam be relatively unso- 
phisticated, and do not require elaborate image under- 
standing algorithms. 

After two dimensional detectiorn of infrared signal 
sources is completed, the frames frosm multiple cameras 
25 can be spatially multiplexed usimg conventional im- 
age processing techniques to derive) a three dimension- 
al spatial localization of each infrarred signal source in 
room 12. To maximize coverage amd ensure three di- 
mensional localization, cameras 25 aare arranged so that 
some combination of at least two canmeras have an over- 
lapping field of view on every part of rroom 12. Each cam- 
era 25 can be calibrated to allow forr spatial localization 
through the use of reference objects; or camera platform 
control. For example, a fixed camerra 20 having a fixed 
focal length is calibrated to allow forr computation of the 
angle of two rays entering its lens biased on location of 
two spots in its image. As will be appreciated by those 
skilled in the art, this angle can be cosmputed from a pub- 
lished focal length of the lens. If neecessary, limited ex- 
periments can be undertaken to determine an exact 
mapping that compensates for difffering focal lengths 
and lens distortions. Calibration continues by providing 
appropriate infrared signal sources sat a known distance 
from a positioned camera that are (.used as permanent 
or semi-permanent reference sources for allowing com- 
putation of ray angles. Typically, thesse reference sourc- 
es are located at junctions of a roorm (for example, tags 
40 and 42 can be used). A new object (such as, for ex- 
ample, infrared tag 41 ) can be locatted provided that at 
least one camera can provide an irmage of the new ob- 
ject and two reference sources, whiMe a second camera 
can image the new object and at Heast one reference 
source. With additional infrared reference sources de- 
tectable by multiple cameras, it is ewen possible to ex- 
tend the method so that camera location with respect to 
the reference sources need not initially be known. 

A wide variety of infrared signal I sources 45 can be 
used for camera calibration, spatial localization, identi- 
fication, and data transfer. For purpcoses of the present 
invention, the infrared signal sourcess 45 can be concep- 
tually divided into reflective tagging] systems that tran- 
siently mark a region or object with liight spots 47, active 
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infrared tags that internally generate infrared light 52 (e. 
g., infrared tags 50 or infrared data transmitter 57 for 
identification of books 55 as seen in Figures 1 and 2), 
or passive infrared tags 51 that controllably reflect inci- 
dent infrared light 54 in response to incident infrared 
light 32 provided by an infrared light source 30. 

Reflective tagging systems use laser pointers (e.g. 
pointer 49 of Figure 1 , held by a user positioned outside 
of the Figure) to direct light spots 47 against predefined 
regions or areas of a room. Typically, infrared lasers are 
used, although visible light lasers can be used in con- 
junction to provide pointing assistance to a user. Visible 
light lasers can even be used alone, at the cost of in- 
creased image proccessing requirements to distinguish 
the visible light signal from the visible background. 

In operation, a laser pointer 49 can be utilized for 
preliminary mapping or surveying operations. The laser 
pointer 49 is pointed at a particular spot on a wall or 
object, and a user pulls a trigger to modulate the directed 
infrared light. Reflected light from the light spot 47 is de- 
tected by cameras 25, and used as a one-time identifi- 
cation point. Each pull of a trigger generates a new light 
modulation pattern, to provide a unique address for the 
identified spatial location. After image processing of the 
many infrared signal sources, a three dimensional map 
(of room 1 2, for example) can be generated by computer 
60. In certain embodiments, it is even possible to use 
coordinating software to direct a user to point to certain 
key portions of room 1 2. For example, a user can be 
directed to point the laser pointer 49 to each corner of 
the room, pulling a trigger to transiently provide a unique 
modulated infrared signal source at each room comer. 
Using only the room corner information, and relying one 
the assumption that the room has rectangular dimen- 
sions, it is possible to determine the 3-dimensional size 
of room 1 2, or any other room of interest. As another 
example, if a whiteboard or other temporary object that 
is relatively immovable is brought into a room, a user 
could direct the laser pointer against the object to define 
its position without bothering to attach an active or pas- 
sive infrared tag. In other applications, the identifying 
spot can be used to guide movable cameras (e.g. cam- 
era 24) to point toward selected regions, objects, or per- 
sons (e.g. desk 16). 

Infrared laser surveying techniques can also assist 
in the development of three-dimensional models to aid 
synthesis of camera views. For example, if remote at- 
tendees to a meeting desire to have a customized view- 
point, or need to edit a prerecorded presentation to in- 
clude camera angles not available in the original pres- 
entation, input from multiple cameras can be synthe- 
sized to interpolate a novel camera view. In Figure 1 , an 
automated surveying tool 80 positioned on desk 16 is 
used to sweep an intermittently generated infrared 
beam 82 along a line 84, or in a regular two dimensional 
pattern. The surveying tool 80 is similar in construction 
and function to pointer 49, but has an additional rotating, 
tilt, or sweep mirror to automatically scan the beam 82 



across a wall or other surface. This scanning creates a 
set of modulated identification spots; 86 having detecta- 
ble identifying reflections that can toe used for camera 
viewpoint synthesis. Alternatively saich identifying spots 

5 can be used to guide cameras or cother instrument to 
point toward specified areas, or to sc:an or patrol an area 
defined by the scan line 84. 

In contrast to reflective tagging, 'which is best suited 
for preliminary surveying use, or fo>r transient tracking 

10 and identification of regions or objeccts, both active and 
passive infrared tags are suited for- long term tracking 
and identification of movable objeccts. Active infrared 
tags are generally larger and more expensive than pas- 
sive infrared tags, requiring a battery or other power 

is source, an infrared emitter such as ain infrared LED, and 
a suitable digital controller. For exaimple, as seen with 
reference to Figure 7, an active infrared tag 110 can be 
built by interconnecting four conventional and widely 
available modules, including a buff ear 112 with IR trans- 

20 mitter LED 1 22, an amplifier 1 1 4 witm I R detector 1 24; a 
microcontroller 116; and a trigger ciircuit 118. A lithium 
battery, photoelectric cell, or other lorng life power source 
120 supplies a low voltage power source for driving the 
modules. In the default state moduleas 112,114, and 1 1 6 

25 are held in a power-down mode. The fourth module, the 
trigger circuit 118, is always active •■ but is designed to 
operate with a very small power-consumption. When the 
trigger circuit 118 is activated by an external signal 1 30 
such an infrared or optical pulse, the modules 1 1 2, 1 1 4, 

30 and 116 are powered. Addressing ssignals 131 may be 
received at module 114, decoded toy module 116 and 
then a response signal 1 32 sent bacck using transmitter 
LED 122 from module 112. The microcontroller module 
116 keeps track of time and after sot>me number of milli- 

35 seconds or the lack of receiver actiwity, will return itself 
(module 116), along with modules 1112 and 114, to the 
power-down state. The response ssignal 132 (infrared 
pulses) incorporates the identity of : the active infrared 
tag, along with any desired data. 

40 As will be appreciated, many differing types of trig- 
ger circuit can be employed. A simpple implementation 
of a trigger circuit include a low-power astable-oscillator 
with a long and slightly random periocd (to avoid repeated 
tag-signal collisions). The trigger ciircuit could also be 

45 designed to be activated by a particularly intense IR 
flash or it might use another mediurm such as the recep- 
tion of a particular radio frequency. In certain contem- 
plated embodiments, the trigger circtuit can be designed 
to render the IR detection/amplificattion module 114 un- 

50 necessary. 

Passive infrared tags 51 are am inexpensive alter- 
native to active infrared tags 50. Passive infrared tags 
51 controllably reflect infrared light: 32 provided by an 
infrared light source 30. The infraredi light source 30 can 

55 be continuously, intermittently, or periodically operated 
as required. In the illustrated embocdiment, the passive 
infrared tags 51 each include an infrrared reflecting ma- 
terial covered by an alternately light ttransmissive or light 
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absorptive shutter. Typically, the shutter is an electrically 
controlled liquid crystal display (LCD) that normally does 
not transmit substantial amounts of infrared light. A low 
power electrical signal is applied to convert the shutter 
from this infrared non-transmissive state to a substan- s 
tially transmissive state. By appropriately switching be- 
tween an infrared light transmissive and non-transmis- 
sive state, information can be coded in the pattern of 
infrared reflections 54 from infrared tags 51 that are de- 
tected by cameras 25. 10 

As will be appreciated by those skilled in the art, 
various activation, data transmission, and timing modi- 
fications to both infrared light source 30 and passive in- 
frared tags 51 are possible to enhance reliability of in- 
formation transfer and conserve power. Using tech- is 
niques similar to that already discussed in connection 
with active infrared tags 50, the passive infrared tags 51 
can be activated to transmit identification codes and oth- 
er data in response to an infrared trigger signal. This 
avoids the need for continuous activation of the LCD 20 
shutter mechanism and greatly reduces long term pow- 
er requirements. As with the active infrared tags 50, mul- 
tiple passive tags can be simultaneously operated, 
since the spatial localizing camera system of the present 
invention can unambiguously distinguish between mul- 25 
tiple passive tags 51 transmitting information in re- 
sponse to the same activation signal. 

Various information transmission and signaling 
schemes are suitable for use in the present invention. 
One preferred scheme, indicated with particular refer- 30 
ence to Figures 3 and 4, can be employed by reflective 
tag (laser pointer 49), an active infrared tag 50, or pas- 
sive infrared tag 51 , as desired. Figure 3 is a graph il- 
lustrating IR pulse detection versus time, with three dis- 
tinct modes of pulsed operation shown. Operation of a 35 
tag 50 or 51 in locating or initialization mode is repre- 
sented by a series of detected periodic infrared pulses 
1 04 identified by bracket 1 06. To conserve power, these 
pulses 1 04 may actually consist of multiple brief infrared 
spikes or pulses, at a rate high enough to ensure appar- 40 
ently continuous detection of an infrared signal by the 
cameras throughout each pulse 104. Of course, IR in- 
tensity can be continuously maintained, rather than 
pulsed, for the duration of each pulse 104 if power is not 
limited. 45 

The periodic detected infrared pulses 104 allow for 
detection of the three dimensional location of a tagged 
region or object (e.g. room walls or books 55) by cam- 
eras 25 and determination of pulse separation 103 be- 
tween sequential pulses 104. As will be appreciated, to so 
prevent aliasing errors and accurately determined pulse 
separation, cameras 25 are operated at a frame rate 102 
substantially faster than pulse separation 103, with a 
frame rate two to three times as fast as pulse separation 
being suggested. 55 

After a brief time (less than a second) periodic blink- 
ing is stopped, and transfer of identification information 
and data commences. Data transfer through blinking of 



tags 50 or 51 is identified by bracken 108. Absence of a 
pulse 104 is interpreted as a binary "0", while presence 
of a pulse 104 is interpreted as a biinary "1". As will be 
appreciated, this allows for information encoding 
through any number of binary encoxiing schemes. For 
best results, use of one or more of the many available 
error correcting codes are preferredl. After identification 
information and data is sent (after possible multiple re- 
sendings), the infrared pulses 104 from tags 50 or 51 
can be stopped as indicated by braclket 110 to conserve 
power. The tags can of course be reactivated at prede- 
termined times, random times, or in response to activa- 
tion signals as desired. 

Detection by cameras 25 of infrared pulses to allow 
for spatial localization and information transfer is sche- 
matically indicated in Figure 4. Two ; sequences of proc- 
essed image frames 122 (camera 11) and 124 (camera 
2) are illustrated. The cameras have a partially overlap- 
ping field of view with three potemtial infrared pulse 
sources. Each frame in the sequenctes 1 22 and 1 24 are 
composite images of several frames, with non infrared 
background visual information subtracted to isolate the 
infrared pulses. 

In frame 1 30 of camera 1 , potemtial positions 1 50, 
151, and 152 of infrared pulses (represented by an as- 
terisk) are indicated by dotted outlime. These positions 
correspond to potential positions 160, 161, and 162 in 
frame 170 of a differently positioned camera 2. Using 
image processing techniques suchi as previously de- 
scribed, the infrared pulses are isollated in frames 122 
and 124 of camera 1 and camera 2! and used as refer- 
ence points. Two dimensional information from each 
camera is acquired, and merged writh calibration infor- 
mation to derive the three dimensiconal position of the 
infrared pulse. 

The three distinct modes of piulsed operation are 
shown in the frames 122 and 124 Ofperation of an infra- 
red tag 50 or 51 in locating or initialization mode is rep- 
resented by periodic infrared pulsees at position 150. 
With each composite and image; processed frame 
130-138 representing a single puis© separation (corre- 
sponding to pulse separation 103 off Figure 3), an infra- 
red pulse is seen at position 1 50 in each frame 1 30-1 38 
of frame sequence 122. A correspomding infrared pulse 
is of course also seen at position 1 600 in frame sequence 
124 for camera 2. This is equivalent! to the series of pe- 
riodic infrared pulses 104 identified! by bracket 106 in 
Figure 3. 

Information transfer from anotther infrared tag is 
shown through aperiodic blinking otf infrared pulses at 
position 151 in frames 130-138 of frrame sequence 122 
(and the corresponding position 1611 in frame sequence 
124). Absence of an infrared pulse in a frame is inter- 
preted as a binary "0" (e.g. frames 11 31 , 1 32, and 1 37), 
while presence of an infrared pulse) is interpreted as a 
binary "1 " (e.g. frames 1 30, 1 33, 1 35/, 1 36, 1 38). Accord- 
ingly, as illustrated in Figure 4, thie binary sequence 
"1001 ... 1101" can be determinecd. This binary se- 
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quence can be header information, data, identification, 
packet control information, error correction information, 
or any other necessary information that might be trans- 
mitted from tags 50 or 51 . 

As previously noted, after appropriate identification 
information and data is sent, the infrared pulses can be 
stopped to conserve power. The tags can of course be 
reactivated to again identify location of tags and transfer 
identification information and other data. In Figure 4, this 
is illustrated by the pattern of infrared pulses at position 
152 in frame sequence 122 (and the corresponding po- 
sition 162 in frame sequence 124). To conserve power, 
no infrared pulses are emitted during a time period cov- 
ering frames 130-136. In response to an activation sig- 
nal, a periodic series of infrared pulses is emitted for 
localization and initialization purposes, as seen in 
frames 1 37 and 1 38. 

In operation, the present system allows for tracking 
of multiple stationary objects (e.g. books 55 or desk 16 
with tag 44) or multiple slow moving objects (e.g. a per- 
son having an attached tag, or a moving laptop compu- 
ter 70 with attached tag 72 held by a person, or both). 
With high speed video cameras the present system can 
be modified to provide spatial localization and tracking 
of quickly moving objects such as a pen 75 with tags 76 
and 77, enabling automated handwriting interpretation 
through tracking of pen motion. Advantageously, large 
numbers of objects can be tracked in parallel, without 
requiring use of sophisticated time or frequency multi- 
plexing tracking technique. 

However, without the use of high speed cameras, 
the major limitation of the foregoing embodiment is its 
relatively low data transfer rate. Because bit transfer is 
closely associated with the frame rate of the cameras, 
the relatively low frame rate of widely available low cost 
cameras limits the theoretical data transfer rate to 1 0 or 
20 bits per second. In practice, because of the neces- 
sary overhead associated with error control, bit transfer 
rates will be even lower. One possible way of overcom- 
ing this problem is the use of secondary high speed data 
communication channels for data transfer, while low da- 
ta transfer rate infrared pulse/camera detection systems 
are used for spatial localization. One example of a dual 
communication channel. system is illustrated in Figures 
5 and 6. 

Figure 5 is a graph illustrating IR pulse detection 
versus time for two separate infrared communication 
channels. Infrared pulses 204 of communication chan- 
nel 1 are detected by cameras operating at a frame rate 
202, using processes described in connection with Fig- 
ures 3 and 4. A separate, higher speed infrared com- 
munication channel 2 with much shorter infrared pulses 
is detectable by a separate (non-camera based) infrared 
communication system. In preferred embodiments, in- 
frared pulses in accordance with applicable IRDA (In- 
frared Data Association) standards can be used for data 
transfer on channel 2, although any high speed commu- 
nication channel, including radio or optical messaging, 



can be used. In operation, the infrared pulses of channel 
1 are used for spatial localization of ain infrared tag, while 
a time coincident communication om channel 2 provides 
high speed data transfer. For exampsle, in Figures 1 and 

5 2, a high speed infrared communication system operat- 
ing at about 19.2 kilobits per second (includes an infrared 
tag 57 and a high speed infrared dcetector 64 attached 
to computer 60. Spatial localization iis provided by cam- 
eras 25 detecting low speed infrareed pulses, while any 

J0 time coincident data received on the I high speed channel 
is associated with a tag at the identified spatial location. 
Of course, use of time coincident cormmunication results 
in a somewhat reduced data throughput due to statisti- 
cally determinable data collisions ((when two or more 

*s tags simultaneously transmit data, rresulting in destruc- 
tive data overlap), but is adequate fror most situations. 

Time coincident data transfer rmethods in accord- 
ance with the present invention cam be better under- 
stood with reference to Figure 6. Tw/o image processed 

20 frame sequences 222 and 224 takeni from different cam- 
eras are illustrated, along with an example of high speed 
data (boxes 270, 272, 274, and 2766) time coincidently 
received with the indicated framess. Data can be re- 
ceived from three separate tags loccated at three differ- 

25 ent spatial positions (positions 250, 1251 , and 252 frame 
sequence 222, positions 260, 261 , amd 262 in frame se- 
quence 224). Using previously disccussed techniques, 
an infrared pulse (denoted by an asUerisk) can be posi- 
tively located in three dimensions griven the two dimen- 

30 sional frame sequences 222 and 2224. If possible, the 
spatial location data is correlated wvith the high speed 
data to provide spatial localization off a specifically iden- 
tified tag. 

Figure 6 illustrates several possible outcomes for 

35 time coincident spatial localization rmethods in accord- 
ance with the present invention. Firame 230 shows a 
spatially localizable infrared pulse att position 250. Time 
coincident data 270 is attributed to> a tag emitting the 
infrared pulse position at position 2150, assuming there 

40 js no data collision. Data collision c:an occur when two 
or more data packets are simultaneously received, or 
when the spatially localizable infrarred pulses overlap. 
Frame 231 illustrates one situation! without data colli- 
sion, with an infrared pulse at positicon 252 being corre- 

45 lated with identification data 272. I However, if two or 
more tags are active during the saime time period, as 
seen in frame 232, the received signsal 274 is the garbled 
result of a data packet collision, arnd the signal is ig- 
nored. After preprogrammed randcom delays (seen in 

so frame 233) or active requests to retransmit, the tags are 
again activated, with hopefully no>n -overlapping data 
transfer to allow for unique identification. Note that in 
certain cases of infrared pulse overrlap, a data collision 
does not necessarily occur, and it rmay be possible to 

55 link spatial location and data throucgh consideration of 
previously received infrared spatial I locations. 

While the present invention hass been described in 
conjunction with specific embodimeents thereof, it is ev- 
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ident that many alternatives, modifications, and varia- 
tions will be apparent to those skilled in the art. Accord- 
ingly, the various embodiments described herein should 
be considered illustrative, and not limiting the scope of 
the present invention as defined in the following claims. 



Claims 

1. A system for virtually modeling a physical system 
having immovable and movable objects, compris- 
ing: 

at least two video cameras, each of the video 
cameras being configured to provide a se- 
quence of images, 

an image processing system configured to ex- 
tract modulated infrared signals from the se- 
quence of images and identify the spatial loca- 
tion of objects, and 

a virtual reality modeling system for construct- 
ing a virtual reality model using identified object 
spatial location information. 

2. The system of claim 1 , wherein a spot of modulated 
infrared light is directed against a predetermined 
object to map its spatial location. 

3. The system of claim 1 or 2, wherein an infrared tag 
is attached to an object, with data relating to the ob- 
ject encoded in the infrared tag. 

4. A system for precisely locating a spot of light, com- 
prising: 

at least two video cameras, with each video 
camera configured to provide a sequence of im- 
ages at a first frame rate, 
a light pointer directing a series of identifying 
light signals at a second rate, less than the first 
frame rate, with a reflection of the identifying 
signals being perceptible by the video camera 
as a modulated light spot, 
an image processing system configured to ex- 
tract a three dimensional spatial location of the 
modulated light spot from the sequence of im- 
ages. 

5. The system of claim 4, wherein the identifying light 
signals are infrared light signals. 

6. The system of claim 5, further comprising a scan- 
ning mechanism for automatically providing a se- 
ries of modulated light spots, with each light spot 
spatially separated from another and having a 
unique identifying light signal. 

7. A system for precisely locating positions of reflec- 



tive surfaces, comprising: 

at least two video cameras, each video camera 
providing a sequence of imaages, 

s a light pointer for directing a . series of identifying 

light signals at a reflective surface, a reflection 
of the identifying signals b(eing perceptible by 
the video camera as a modujlated light spot, and 
an image processing systerm configured to ex- 

10 tract a three dimensional spatial location of the 

modulated light spot from tme sequence of im- 
ages derived from two or rmore cameras. 

8. The system of claim 7, wherein the identifying light 
15 signals are infrared light signalss. 
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